Skip to content

bench: rank file sizes against Parquet in both size groups - #227

Merged
lwwmanning merged 24 commits into
mainfrom
wm/consolidate-file-size-metrics-0b3056
Nov 21, 2026
Merged

lwwmanning merged 24 commits into
mainfrom
wm/consolidate-file-size-metrics-0b3056

Conversation

@lwwmanning

@lwwmanning lwwmanning commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

What changes

This changes bench's size groups, its landing page, and the file size panel on the vortex.dev homepage. Old Compression Size and SQL File Sizes links still work.

Before, Compression Size ranked formats on the compress-bench datasets alone. Now Compression & File Size ranks every format against zstd Parquet on those datasets and on TPC-DS at scale factor 1 (SF1). It uses the newest result of each format, and leaves out DuckDB's database file. In this group only, the two Parquet series show as parquet (zstd) and parquet (snappy).

Before, SQL File Sizes ranked formats against Vortex, counted TPC-H four times, and showed an Arrow column with no values. Now SQL Benchmark File Sizes comes after the query groups and ranks against parquet. It ranks only the benchmarks that write Parquet with bench's settings, and counts each benchmark once.

Before, the homepage panel showed a size score for a format with no TPC-DS sizes. Now it shows "Not measured", as the other panels do, until develop records Lance and Arrow IPC on TPC-DS.

Query summaries rank a series that misses a query below every complete series, so Lance cannot gain by skipping TPC-DS q72. The charts draw Snappy Parquet with dotted lines. The methodology link and build commit move to the top of the landing page, and the methodology page explains the settings of bench's Parquet writer.

The new sizes and the writer settings come from vortex-data/vortex#11720.

How it was checked

bun run check:ci, tsc, the site tests, the bench tests with their Postgres integration suite, the build, and knip pass. No screenshots yet.

  • Every commit is signed off (git commit -s), for the DCO check.
  • bun run check:ci passes.
  • Each new image or other asset has an annotation in REUSE.toml.

🤖 Generated with Claude Code

The Compression Size and SQL File Size groups showed the same metric,
file size per format, from two suites. One File Sizes group now charts
the compress-bench datasets and three SQL scopes: ClickBench, TPC-H at
SF10, and TPC-DS at SF1. Old Compression Size links redirect to it, and
the homepage card reads the group by its new or old name.

The summary compares each format with zstd Parquet on the compress-bench
datasets and on TPC-DS. Each format uses its newest commit that has zstd
Parquet. The wide-table and sparse-wide families count as one dataset
each. A format measured on only some datasets says how many, has no
total, and ranks after the complete formats. Parquet codec variants,
such as parquet-snappy, draw in Parquet's color with a dashed line.

Signed-off-by: Will Manning <will@willmanning.io>
File Sizes moves to the Compression Size group key, so old Compression
Size links open it. SQL File Sizes comes back on its own key: it charts
every SQL benchmark and scale factor and ranks formats against Vortex,
as before. Each group's info tooltip says how the two differ.

Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
A series with no time for a query now ranks below every series that ran
all of them. The missing-series penalty alone scores a skipped query at
2x the best time, so a series could rank first by skipping the slowest
query. Lance skips TPC-DS q72, which it cannot run.

Snappy Parquet joins TPC-DS's queries, so a codec variant is now dotted,
and dash-dotted on DuckDB, to keep it apart from DuckDB's dashed Parquet.

Signed-off-by: Will Manning <will@willmanning.io>
@vercel

vercel Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
vortex.dev Ready Ready Preview Nov 21, 2026 10:27pm UTC

Request Review

@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

🔦 Lighthouse audit

URL Profile Perf A11y Best practices SEO
/ desktop 🟢 100 🟢 100 🟢 100 🟢 100
/ mobile 🟢 95 🟢 100 🟢 96 🟢 100
/bench desktop 🟢 100 🟢 100 🟢 100 🟢 100
/bench mobile 🟢 96 🟢 100 🟢 100 🟢 100
/bench/chart/ct.eyJrIjoiQ29tcHJlc3Npb25UaW1lIiwiZGF0YXNldCI6ImFyYWRlIiwiZGF0YXNldF92YXJpYW50IjpudWxsfQ desktop 🟢 100 🟢 100 🟢 100 🟢 100
/bench/chart/ct.eyJrIjoiQ29tcHJlc3Npb25UaW1lIiwiZGF0YXNldCI6ImFyYWRlIiwiZGF0YXNldF92YXJpYW50IjpudWxsfQ mobile 🟢 99 🟢 100 🟢 96 🟢 100
/bench/methodology desktop 🟢 100 🟢 100 🟢 100 🟢 100
/bench/methodology mobile 🟢 100 🟢 100 🟢 100 🟢 100
/blog desktop 🟢 99 🟢 100 🟢 100 🟢 100
/blog mobile 🟢 97 🟢 100 🟢 100 🟢 100
/blog/btrblocks-compressor desktop 🟢 100 🟢 100 🟢 100 🟢 100
/blog/btrblocks-compressor mobile 🟢 99 🟢 100 🟢 100 🟢 100
/faq desktop 🟢 100 🟢 100 🟢 100 🟢 100
/faq mobile 🟢 99 🟢 100 🟢 100 🟢 100

Run 44956865252 · 2026-11-21 22:29 UTC

Vercel adds its Toolbar to preview deployments, and the proxy's
x-vercel-skip-toolbar header does not stop it. The Toolbar loads from
vercel.live and Pusher, and its third-party cookies and issues cut Best
Practices to 0.78 on every audited page. The site loads neither host,
so Lighthouse now blocks both.

Signed-off-by: Will Manning <will@willmanning.io>
Parquet is zstd level 3, the size and throughput baseline. Parquet
(Snappy) is Parquet as DuckDB writes it by default, which TPC-DS SF1
queries and the compression benchmark sizes. ClickBench and FineWeb query
their published files as Parquet, and ClickBench's are Snappy. The page
also says that a query summary ranks a series that missed a query below
the complete ones, as Lance on TPC-DS q72.

The contract now names file sizes by format directory, not by codec.

Signed-off-by: Will Manning <will@willmanning.io>
vortex-data/vortex#11720 now installs the DuckDB 2.0.0 CLI, the version
the queries run on.

Signed-off-by: Will Manning <will@willmanning.io>
A new section gives each Parquet writer setting and the test behind it:
zstd level, dictionary size, page size and page index, Bloom filters, and
row groups. Parquet (Snappy) is now the same Parquet with Snappy in place
of zstd, rather than Parquet as DuckDB writes it, and TPC-DS at scale factor 1
no longer queries it.

Signed-off-by: Will Manning <will@willmanning.io>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-21T22:30:25.834377Z 65f672f New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

…ize-metrics-0b3056

Signed-off-by: Will Manning <will@willmanning.io>

# Conflicts:
#	lighthouserc.cjs
#	src/bench/lib/methodology.ts
#	src/bench/lib/queries.ts

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c0cccc5a6e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/bench/lib/summary.ts Resolved
Comment thread src/bench/lib/queries.ts Resolved
…cache

A format with one wide table or sparse-wide dataset counted the whole family
as measured, so it could rank in the complete tier against formats measured
on every member. A family now counts as measured only when the format has
every dataset of it that any format has.

The File Sizes group reuses the Compression Size slug with another name,
other charts, and another summary, so the group payload cache moves to v16.

Signed-off-by: Will Manning <will@willmanning.io>
…B's file out of its ranking

TPC-DS records DuckDB's database file, which is a query engine's storage
rather than a file format, so the summary no longer ranks it. The group
still charts it. The contract and methodology also say that TPC-DS SF1
records Snappy Parquet's size.

Signed-off-by: Will Manning <will@willmanning.io>
@lwwmanning lwwmanning changed the title bench: one File Sizes group ranks every format against zstd Parquet bench: one File Sizes & Compression Ratios group ranks every format against zstd Parquet Nov 21, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f5eb6dbbaf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/bench/lib/descriptions.ts Resolved
Comment thread src/bench/lib/file-sizes.ts Resolved
Comment thread src/bench/lib/compression-suite.ts Resolved
Comment thread src/bench/lib/summary.ts Resolved
The new name keeps more continuity with the Compression Size group it
replaces.

Signed-off-by: Will Manning <will@willmanning.io>
@lwwmanning lwwmanning changed the title bench: one File Sizes & Compression Ratios group ranks every format against zstd Parquet bench: one Compression & File Size group ranks every format against zstd Parquet Nov 21, 2026
The group charts Snappy Parquet beside zstd Parquet, so its legends,
tooltips, toolbar chips, and summary card name the zstd files
parquet-zstd, and the card's baseline header reads "vs parquet-zstd".
ClickBench's chart keeps parquet, because it charts the published
Snappy files. The series keys stay parquet, so filters, colors, and
URLs are unchanged. A compression chart's permalink page uses the
same label; a file-size permalink page does not, because the SQL File
Sizes group shares those charts. The group payload cache moves to v17.

Signed-off-by: Will Manning <will@willmanning.io>
…atio

The Compression & File Size card ordered formats by their ratio versus
Arrow once every format had one. TPC-DS has no Arrow size, so that order
left out the one SQL scope the summary ranks. A size summary now orders
by the Arrow ratio only when it covers every dataset each format is
ranked on, and by the Parquet ratio otherwise. The String Encoding Size
card, whose Arrow ratios cover every dataset, keeps its order.

The methodology now says what the vs Arrow column measures, that it
covers the compression datasets alone, and how the card is ordered. It
also says that the compression benchmark writes 1,048,576-row row
groups. The group payload cache moves to v18.

Signed-off-by: Will Manning <will@willmanning.io>
On the landing page, the line with the methodology link and the build
commit was at the bottom, after every group. It now opens the page's
main content, left-aligned with the group cards and small, and the
methodology link is underlined so it reads as a link. The chart and
methodology pages keep the line at the bottom.

Signed-off-by: Will Manning <will@willmanning.io>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f7d36b5b26

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/bench/components/GroupToolbar.tsx Resolved
In the Compression & File Size group, the toolbar's parquet-zstd chip
also toggles ClickBench's parquet line, which is the published Snappy
files. A renamed chip's title now says so: "parquet-zstd, and parquet
on clickbench".

Signed-off-by: Will Manning <will@willmanning.io>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a4c669cecf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/bench/components/Chart.tsx Resolved
Comment thread src/lib/bench-summary.ts Resolved
Rename the SQL File Sizes group to SQL Benchmark File Sizes and move it
after the query groups, since it holds the files those queries read.

Its summary now compares each format with parquet, bench's standard
Parquet, at each format's newest commit that also has parquet. It ranks
only JOB, Polar Signals, statpopgen, TPC-DS, TPC-H, and nested TPC-H,
whose Parquet bench writes with its own settings; ClickBench, FineWeb,
Appian, sorted ClickBench, and the Vortex queries keep their charts.
Each benchmark counts once, as the geometric mean of its scale factors,
so TPC-H's three scale factors no longer outweigh the rest. DuckDB's
database file is left out, as in the Compression & File Size summary.

The card drops the Total and vs Arrow columns, which this summary never
filled, and says "of N benchmarks" in its coverage note. With no
baseline, the landing page's headline returns for this group.

Signed-off-by: Will Manning <will@willmanning.io>
The homepage's file size panel dropped bench's coverage for that one
panel, so a format without TPC-DS sizes, such as Lance today, showed a
geomean over fewer datasets beside formats measured on all of them.
Lance is about 2x Parquet without TPC-DS and 4.3x on it, so the panel
flattered it. Pass the coverage through, as the other panels do;
results.js shows such a row as "Not measured".

Signed-off-by: Will Manning <will@willmanning.io>
…factors

The SQL Benchmark File Sizes card counted a benchmark only when a format
had every scale factor of it, so Lance, with TPC-DS at SF1 alone, showed
a 2.12x score beside "measured on 0 of 6 benchmarks". It now counts one
scale factor as enough, so Lance shows 1 of 6 and still ranks after the
formats measured on every benchmark. The Compression & File Size summary
keeps the stricter rule for its families.

Signed-off-by: Will Manning <will@willmanning.io>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2e0d8959f9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/bench/lib/summary.ts Resolved
DuckDB's database file is the file DuckDB's queries read, so the SQL
Benchmark File Sizes summary keeps it when it has a size on every
benchmark the summary covers. With fewer, it stays out, as in the
Compression & File Size summary.

Signed-off-by: Will Manning <will@willmanning.io>
@lwwmanning lwwmanning changed the title bench: one Compression & File Size group ranks every format against zstd Parquet bench: rank file sizes against Parquet in both size groups Nov 21, 2026
The collapsed group's size headline read the Vortex ratio alone, so a
mean over fewer datasets than the summary covers showed as a plain
"smaller than Parquet" claim while the card marked it partial. The
headline now adds "(partial)" when Vortex's measured count is below
the total.

Signed-off-by: Will Manning <will@willmanning.io>
lwwmanning added a commit to vortex-data/vortex that referenced this pull request Nov 21, 2026
This changes how the benchmarks write Parquet, and which Parquet sizes
they record. ClickBench and FineWeb still query their published files as
`parquet`.

Before, each suite set its own Parquet writer settings. statpopgen and
SpatialBench's zone table wrote uncompressed Parquet, and the `vortex`
suite wrote Snappy. Now every generated Parquet file uses one config, in
`vortex-bench/src/utils/parquet.rs`. It sets zstd level 3, dictionaries
of at most 64 KiB, a page index, and pages of at most 20,000 rows and 1
MiB. Each file records a layout version, and the generators write a
cached file in an older layout again.

Before, TPC-H and nested TPC-H wrote row groups of 1,048,576 rows on all
storage, and TPC-DS wrote 122,880. Now every query suite writes 122,880
rows (DuckDB's default) for local NVMe and 1,048,576 rows (parquet-rs's
default) for S3. In a TPC-H test at scale factor 10, the small row
groups were 28% to 38% faster on NVMe. On S3, the large row groups were
14% to 21% faster on DataFusion, and 3% to 8% slower on DuckDB.

Each other setting was tested on its own, on TPC-H at scale factor 10
and on the random access datasets. 1 MiB dictionaries made files 22%
larger, DataFusion 8% slower, and lookups up to 2.8 times slower. Bloom
filters made files 14% larger and sped up no query. The 20,000-row page
limit made lookups up to 4 times faster at the same size.

compress-bench and TPC-DS at scale factor 1 now also record the size of
`parquet-snappy`: the same Parquet as `parquet`, with Snappy in place of
zstd. No query reads it. TPC-DS at scale factor 1 also records the size
of Arrow IPC and queries Lance on DataFusion. Lance skips q72, which
overflows its 32-bit string offsets.

Parquet sizes and times change for every suite that generates Parquet.
The bench side is vortex-data/vortex.dev#227.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Will Manning <will@willmanning.io>
The Compression & File Size group labeled its two Parquet series
parquet-zstd and parquet-snappy. They now read parquet (zstd) and
parquet (snappy), in the legends, tooltips, chips, and the summary's
"vs" header. ClickBench's chart still shows plain parquet for its
published Snappy files.

A renamed chip's title now lists, for each renamed format, the charts
that keep that format's real name, so the parquet (snappy) chip does
not claim ClickBench charts parquet-snappy.

Signed-off-by: Will Manning <will@willmanning.io>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9fceee30e7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/bench/lib/site-files.ts Resolved
Comment thread src/bench/lib/headline.ts Resolved
…y benchmark

In the Compression & File Size group, the collapsed headline and
llms.txt said "Parquet" although the summary divides by zstd Parquet
and the group also charts Snappy Parquet. They now say "Parquet (zstd)".

llms.txt described the SQL Benchmark File Sizes score as a geometric
mean over datasets, and its range as across datasets. The score now
says each benchmark counts once, as the geometric mean of its scale
factors, and the range is across benchmarks and scale factors.

Signed-off-by: Will Manning <will@willmanning.io>
@lwwmanning
lwwmanning merged commit e4cec6a into main Nov 21, 2026
32 checks passed
@lwwmanning
lwwmanning deleted the wm/consolidate-file-size-metrics-0b3056 branch November 21, 2026 22:51

This branch was successfully deployed

1 active deployment
Preview — 65f672fd Deployed Nov 21, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant